Skip to content

Add CTRF export format for reports (admin + WP-CLI)#1271

Merged
davidperezgar merged 6 commits intotrunkfrom
1270-add-ctrf-export-format
May 1, 2026
Merged

Add CTRF export format for reports (admin + WP-CLI)#1271
davidperezgar merged 6 commits intotrunkfrom
1270-add-ctrf-export-format

Conversation

@davidperezgar
Copy link
Copy Markdown
Member

@davidperezgar davidperezgar commented Apr 22, 2026

What?

Closes #1270

Add CTRF export support for Plugin Check reports in both admin exports and WP-CLI output.

Why?

Plugin Check already supports CSV/JSON/Markdown exports, but lacks a standardized test report format for CI tooling and platforms that consume CTRF.

This PR adds first-class CTRF output so results can be integrated with the CTRF ecosystem.

How?

  • Added CTRF format support to the exporter:
    • New Results_Exporter::FORMAT_CTRF constant.
    • Added CTRF payload generation with required schema fields:
      • reportFormat: "CTRF"
      • specVersion: "1.0.0"
      • results.tool.name: "plugin-check"
      • results.summary + results.tests[]
    • Refactored CTRF generation into helper methods to satisfy PHPMD complexity rules.
    • Updated CTRF export filename extension to .ctrf.json.
  • Added admin export support:
    • Added Export CTRF localized string.
    • Added CTRF export button in admin JS (format: ctrf).
  • Added CLI support:
    • Added ctrf and strict-ctrf to accepted --format options.
    • For both formats, output is strict JSON CTRF (no FILE: headers), while preserving existing filtering behavior.
  • Updated documentation:
    • docs/CLI.md now documents ctrf and strict-ctrf and includes CLI example.
    • readme.txt changelog updated.
  • Added tests:
    • New PHPUnit tests for CTRF export payload and empty-results behavior.
    • Extended Behat scenario to verify --format=ctrf and --format=strict-ctrf.

Testing Instructions

  1. Run coding standards checks: 
    composer phpmd
./vendor/bin/phpcs includes/Utilities/Results_Exporter.php tests/phpunit/tests/Utilities/Results_Exporter_Tests.php
  2. Verify CLI docs include the new formats: 
    rg -n "ctrf|strict-ctrf" docs/CLI.md includes/CLI/Plugin_Check_Command.php
  3. In wp-admin Plugin Check screen, run checks and confirm export buttons include Export CTRF.
  4. Test CLI output formats: 
    wp plugin check <plugin> --format=ctrf
wp plugin check <plugin> --format=strict-ctrf
    Confirm output is valid JSON and contains "reportFormat": "CTRF".
  5. Confirm exported CTRF filename ends with .ctrf.json.

AI Usage Disclosure

  • This PR was created without the help of AI tools
  • This PR includes AI-assisted code or content

If AI tools were used, please describe how they were used:
Used Codex/ChatGPT to draft and implement code changes, tests, and documentation updates based on issue requirements.

@davidperezgar davidperezgar linked an issue Apr 22, 2026 that may be closed by this pull request
Add CTRF export format for Plugin Check reports (ctrf) #1270
Closed
6 tasks
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 22, 2026 •
edited
Loading

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: davidperezgar <davidperez@git.wordpress.org>
Co-authored-by: Luc45 <lucasbustamante@git.wordpress.org>
Co-authored-by: ernilambar <nilambar@git.wordpress.org>
Co-authored-by: frantorres <frantorres@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

Luc45
Luc45 suggested changes Apr 28, 2026
View reviewed changes
Copy link
Copy Markdown

@Luc45 Luc45 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Howdy David, I passed the results through a CTRF validator using ctrf-cli and it failed one small thing:

  1. Check out this PR in wp-env, install a plugin with findings (e.g. wp plugin install hello-dolly --activate), then generate a CTRF report:

    wp plugin check hello-dolly --format=ctrf > report.json

  2. Validate it:

    npx --package=ctrf-cli@0.0.5 -- ctrf-cli validate report.json

  3. Output:

    ✗ report.json failed validation:
  - /results/tests/0: must NOT have additional properties
  - /results/tests/1: must NOT have additional properties
  ...

The culprit is tests[].labels — it isn't in the CTRF spec, and the test-item schema is additionalProperties: false. CTRF reserves extra (object) as the official extension point ("Extension point for arbitrary metadata"), so renaming labels → extra in Results_Exporter::build_ctrf_test() (plus the matching assertions in Results_Exporter_Tests.php) should be enough. I confirmed locally — after that rename both validate and validate-strict pass clean.

@Luc45
Copy link
Copy Markdown

Luc45 commented Apr 28, 2026

I do wonder, though, that ideally this should be a PR to PHPCS.

For now, I'm converting PHPCS to CTRF in my tool, so Plugin Check being able to generate CTRF isn't a blocker for me.

@Luc45
Copy link
Copy Markdown

Luc45 commented Apr 28, 2026

I've opened a PR to PHPCS: PHPCSStandards/PHP_CodeSniffer#1420

If they merge this, we could update the PHPCS binary here and use the upstream CTRF reporter instead (since I'll be using CTRF for other PHPCS checks other than plugin check, too)

@davidperezgar
Copy link
Copy Markdown
Member Author

davidperezgar commented Apr 28, 2026

We could make the conversion in this repository.

@Luc45
Copy link
Copy Markdown

Luc45 commented Apr 28, 2026

Yes - so, as discussed in DM, I've pushed CTRF reporter for PHPCS as a library: https://github.com/Luc45/phpcs-ctrf-reporter

Your call if you want to use it or integrate something in PCP directly.

@Luc45
Copy link
Copy Markdown

Luc45 commented Apr 28, 2026

As part of validating that reporter, I used it against WooCommerce and compared the integrity of the CTRF output against a native PHPCS output and they match, both in serial mode or in parallel, passing strict validation by ctrf-cli.

This, together with the unit/integration tests it has, gives me confidence that it's in a good shape.

ernilambar
ernilambar reviewed Apr 29, 2026
View reviewed changes
Comment thread includes/Utilities/Results_Exporter.php Outdated
@ernilambar
Copy link
Copy Markdown
Member

ernilambar commented Apr 29, 2026 •
edited
Loading

I tested the PR locally and confirm that replacing "labels" to "extra" will generate valid CTRF report.

@davidperezgar davidperezgar merged commit 523f062 into trunk May 1, 2026
29 checks passed
@davidperezgar davidperezgar deleted the 1270-add-ctrf-export-format branch May 1, 2026 15:48
@davidperezgar davidperezgar added this to the 2.0.0 milestone May 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ernilambar ernilambar ernilambar approved these changes

@frantorres frantorres frantorres approved these changes

+1 more reviewer

@Luc45 Luc45 Luc45 requested changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

2.0.0

Development

Successfully merging this pull request may close these issues.

Add CTRF export format for Plugin Check reports (ctrf)

4 participants

@davidperezgar @Luc45 @ernilambar @frantorres